Apparatus and Methods for Encoding, Decoding and Representing High Dynamic Range Images

ABSTRACT

A data structure defining a high dynamic range image comprises a tone map having a reduced dynamic range and HDR information. The high dynamic range image can be reconstructed from the tone map and the 
     HDR information. The data structure can be backwards compatible with legacy hardware or software viewers. The data structure may comprise a JFIF file having the tone map encoded as a JPEG image with the HDR information in an application extension or comment field of the JFIF file, or a MPEG file having the tone map encoded as a MPEG image with the 
     HDR information in a video or audio channel of the MPEG file. Apparatus and methods for encoding or decoding the data structure may apply pre- or post correction to compensate for lossy encoding of the high dynamic range information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/488,488, filed on Sep. 17, 2014, which is a continuation of U.S.application Ser. No. 13/480,151, filed on May 24, 2012, which is acontinuation of U.S. application Ser. No. 11/831,709, filed on Jul. 31,2007, now U.S. Pat. No. 8,514,934, and which is a continuation of U.S.application Ser. No. 11/568,030, filed on Jan. 7, 2008, now U.S. Pat.No. 8,218,625, which is a national stage application of PCT/CA2004/02199filed on Dec. 24, 2004 which claims the benefit of U.S. Application No.60/564,608 filed on Apr. 23, 2004. The entire contents of each of theforegoing applications is incorporated herein by reference.

TECHNICAL FIELD

The invention relates to high dynamic range digital images. Theinvention relates specifically to methods and apparatus for encoding anddecoding high dynamic range images and to data structures containingdigital high dynamic range images.

BACKGROUND

Human vision is capable of appreciating contrast ratios of up to1:10,000. That is, a person can take in a scene in which some parts ofthe scene are 10,000 times brighter than other parts of the scene andsee details in both the brightest and darkest parts of the scene.Further, human vision can adapt its sensitivity to brighter or darkerscenes over a further 6 orders of magnitude.

Most conventional digital image formats (so-called 24-bit formats) useup to 24 bits to store color and luminance information for each pixel inan image. For example, each of a red, green and blue (RGB) value for apixel may be stored in one byte (8 bits). Such formats are capable ofrepresenting brightness variations over only about two orders ofmagnitude (each byte can store one of 256 possible values). There exista number of standard formats for representing digital images (whichinclude both still and video images). These include JPEG (JointPhotographic Experts Group), MPEG (Motion Picture Experts Group), AVI(Audio Video Interleave), TIFF (Tagged Image File Format), BMP (BitMap), PNG (Portable Network Graphics), GIF (Graphical InterchangeFormat), and others. Such formats may be called “output referredstandards” because they do not attempt to preserve image informationbeyond what can be reproduced by electronic displays of the types mostcommonly available. Until recently, displays such as computer displays,televisions, digital motion picture projectors and the like have beenincapable of accurately reproducing images having contrast ratios betterthan 1:1000 or so.

Display technologies being developed by the assignee, and others, areable to reproduce images having high dynamic range (HDR). Such displayscan reproduce images which more faithfully represent real-world scenesthan conventional displays. There is a need for formats for storing HDRimages for reproduction on these displays and other HDR displays thatwill become available in the future.

A number of formats have been proposed for storing HDR images as digitaldata. These formats all have various disadvantages. A number of theseformats yield prohibitively large image files that can be viewed onlythrough the use of specialized software. Some manufacturers of digitalcameras provide proprietary RAW formats. These formats tend to becamera-specific and to be excessive in terms of data storagerequirements.

There is a need for a convenient framework for storing, exchanging, andreproducing high dynamic range images. There is a particular need forsuch a framework which is backwards compatible with existing imageviewer technology. There is a particular need for backwardscompatibility in cases where an image may need to be reproduced bylegacy devices, such as DVD players, which have hardware-based imagedecoders.

SUMMARY OF THE INVENTION

One aspect of this invention provides methods for encoding high dynamicrange image data. The methods involve obtaining tone map datacorresponding to the high dynamic range image data. The tone map datahas a dynamic range lower than that of the high dynamic range imagedata. The method computes ratio data comprising ratios of values in thehigh dynamic range image data and corresponding values in the tone mapdata; generates high dynamic range information based on the ratio data;generates tone map information based on the tone map data; and, storesthe high dynamic range information and the tone map information in adata structure.

The data structure may be readable by legacy image viewers. The legacyimage viewers may read the tone map information and ignore the highdynamic range information. In some embodiments, the data structurecomprises a JFIF file and the tone map information comprises a JPEGimage. In some embodiments, the data structure comprises a MPEG file andthe tone map information comprises a frame of a MPEG video.

Another aspect of the invention provides a data structure forrepresenting a high dynamic range image having an initial dynamic range.The data structure comprises a tone map portion and a high dynamic rangeinformation portion. The tone map portion contains tone map informationrepresenting the image and has a dynamic range less than the initialdynamic range. The high dynamic range information portion containsinformation describing ratios of luminance values in the tone mapportion to luminance values of the high dynamic range image.

Another aspect of the invention provides apparatus for encoding highdynamic range images.

Further aspects of the invention and features of specific embodiments ofthe invention are described below.

BRIEF DESCRIPTION OF THE DRAWINGS

In drawings which illustrate non-limiting embodiments of the invention,

FIG. 1 is a data flow diagram illustrating a method for creating a HDRimage file according to a general embodiment of the invention;

FIG. 2 is a flow chart giving an overview of HDR image encoding anddecoding methods according to the invention;

FIG. 3 is a data flow diagram illustrating a method for creating a HDRimage file according to one specific embodiment of the invention;

FIG. 4 is a flow chart illustrating methods according to someembodiments of the invention which provide corrections for artefactsresulting from compression and/or downsampling; and,

FIG. 5 is a flow chart illustrating a method according to one embodimentof the invention which provides corrections for artefacts resulting fromcompression and/or downsampling during reconstruction of a HDR image.

DESCRIPTION

Throughout the following description, specific details are set forth inorder to provide a more thorough understanding of the invention.However, the invention may be practiced without these particulars. Inother instances, well known elements have not been shown or described indetail to avoid unnecessarily obscuring the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative, ratherthan a restrictive, sense.

One aspect of this invention provides data structures for representingHDR images (HDR data structures). In preferred embodiments, the HDR datastructures permit images to be viewed in a standard dynamic range modeusing standard image viewing software and permits high dynamic rangeversions of the same images to be viewed in a high dynamic range modeusing a HDR viewer and appropriate HDR display.

FIG. 1 shows a system 10 according to the invention for creating HDRdata structures 16 and for viewing images represented by HDR datastructures 16. FIG. 2 shows a method 30 performed by system 10 forcreating HDR data structures and alternative methods 31A and 31B fordisplaying images from data in the HDR data structures 16.

System 10 comprises an encoder 14 for creating a HDR image datastructure 16 based on original HDR image data 12. Data structure 16 maybe decoded by a standard decoder 18 to provide a standard dynamic rangeimage 19. In some embodiments of the invention Standard decoder 18comprises a “legacy” hardware decoder or software-based decoder such assuitable image viewer software. Data structure 16 may be decoded by aHDR decoder 20 to yield a reconstructed HDR image 21.

Method 30 begins in block 32 by acquiring HDR image data 12. HDR imagedata 12 includes information that directly or indirectly specifies theluminance of pixels in an image. HDR image data 12 may be in anysuitable format and may be acquired through the use of a suitable HDRcamera (possibly by combining multiple exposures) or rendered directlyin a computer. The source of HDR image data 12 is not important to thepractice of the invention.

Method 30 also obtains (block 34) tone map data 15 corresponding to HDRimage data 12. Tone map data 15 represents a likeness of the image ofHDR image 12, but has a lower dynamic range than HDR image data 12. Tonemap data 15 may be generated from HDR image data 12, as indicated byline 13, or in some other manner derived from data having a commonsource with HDR image data 12. If tone map data 15 is not derived fromHDR image data 12 then the order in which tone map data 15 and HDR imagedata 12 are obtained (i.e. the order of blocks 32 and 34) is notimportant.

Encoder 14 generates a data structure 16. Data structure 16 includes atone map portion 16A, which is based on tone map data 15, and an HDRinformation portion 16B, which contains information that may be combinedby HDR decoder 20 with the data from tone map portion 16A to reconstructHDR image data 12 or a close approximation thereto. Method 30 generates(block 36) HDR information portion by comparing tone map data 15 (or,equivalently, tone map data reconstructed from tone map portion 16A ofdata structure 16) and HDR image data 12. In block 38, method 30 storestone map portion 16A and HDR information portion 16B in data structure16.

In preferred embodiments of the invention, data structure 16 has aformat that can be read by a standard decoder to produce a lower dynamicrange (LDR) image. A standard decoder 18 may implement decoding method31A. Standard decoder 18 generates a standard LDR image 19 by retrievingtone map portion 16A and displaying an image represented by tone mapportion 16A (block 39). The standard decoder can ignore HDR informationportion 16B.

Data structure 16 can also be read by an HDR decoder 20. HDR decoder 20implements decoding method 31B and generates an HDR image 21 based uponinformation from both tone map portion 16A and HDR information portion16B. Method 31B retrieves data from tone map portion 16A and HDRinformation portion 16B of data structure 16 in block 40. In block 42, areconstructed HDR image is created by modifying a tone map extractedfrom tone map portion 16A according to HDR information from HDRinformation portion 16B. The reconstructed HDR image is displayed inblock 44.

Tone map portion 16A may be in any suitable format. For example, tonemap portion 16A may be in JPEG, MPEG, AVI, TIFF, BMP, GIF or some othersuitable format. Tone map portion 16A comprises information thatdirectly or indirectly specifies luminance of pixels in the image with adynamic range that is less than that of original HDR image 12. Where theHDR image data 12 specifies a color image, tone map portion 16Apreferably includes information specifying colors for pixels in theimage.

In some embodiments of the invention, data structure 16 comprises a JPEGFile Interchange Format (JFIF) formatted file. In such embodiments, tonemap portion 16A may be contained in the image portion of a JFIF file andHDR information portion 16B may be stored in one or more applicationextension portions of the JFIF file and/or in one or more commentportions of the JFIF file. In such embodiments any standard JPEG viewercan open data structure 16 and display the image provided in tone mapportion 16A at a dynamic range lower than that of the original HDR data12 or reconstructed HDR image 21.

Standard JPEG viewers ignore application extensions in JFIF files thatthey do not support. Thus, the presence of HDR information portion 16Bhas substantially no effect on the viewing of an image from datastructure 16 using any standard JPEG viewer. Where HDR information 16Bis in a comment field of a JFIF file, HDR information 16B is preferablyencoded as ASCII text since some applications may attempt to readcomment fields of JFIF files. Such applications may expect the commentfields to contain only text and may behave improperly upon attempting toopen a comment field that contains data of an unexpected type. Version1.2 is one version of JFIF. JFIF version 1.2 is fully described in AnnexB of ISO DIS 10918 1, which is hereby incorporated herein by reference.

In some embodiments of the invention, data structure 16 comprises a MPEGformatted file. In such embodiments, tone map portion 16A may becontained in the image portion of a MPEG file and HDR informationportion 16B may be stored in one or more application extensions the MPEGfile and/or in one or more comment portions of the MPEG file. In suchembodiments any standard MPEG viewer can open data structure 16 anddisplay the image provided in tone map portion 16A at a dynamic rangelower than that of the original HDR data 12 or reconstructed HDR image21. One HDR information portion 16B may be associated with each frame ofa MPEG video file, or for versions of MPEG which employ keyframes, HDRinformation portion 16B may be associated only with the keyframes.Conventional MPEG keyframe interpolation techniques may be used tocreate the inter-keyframe frames (i.e., the frames which are between thekeyframes).

Standard MPEG viewers ignore channels of MPEG files that they do notsupport. Thus, the presence of HDR information portion 16B hassubstantially no effect on the viewing of an image from data structure16 using any standard MPEG viewer. Where HDR information 16B is in acomment field of a MPEG file, HDR information 16B is preferably encodedas ASCII text since some applications may attempt to read comment fieldsof MPEG files. Such applications may expect the comment fields tocontain only text and may behave improperly upon attempting to open acomment field that contains data of an unexpected type.

Tone map portion 16A may be created from tone map data 15 in anysuitable manner. For example, tone map portion 16A may be generatedusing a suitable tone mapping operator. The tone mapping operatorpreferably has the properties that:

-   -   an original HDR input (i.e. original HDR image data 12) is        mapped smoothly into a standard dynamic resolution (typically        24-bit) output domain;    -   no components of the output of the tone mapping operator are        clamped at values of 0 or 255;    -   hue is maintained for each pixel; and,    -   if the tone mapping operator changes saturation values, it makes        only mild changes that may be described by invertible functions.        The inventors have found that the bilateral filter described in        Durand and Dorsey, Fast bilateral filtering for the display of        high dynamic range images, ACM Transactions on Graphics, 21, 3,        249-256 (2002) provides a suitable tone mapping operator. Tone        map portion 16A may be encoded using a suitable encoder, such as        a JPEG encoder or a MPEG encoder.

Tone map portion 16A may represent pixel color values in any suitablemanner. For example, pixel color values may be represented as RGB (red,green and blue) values, CMYK (cyan, magenta, yellow and black) values,YCbCr (luminance and chrominance) values, or the like. The data in tonemap portion 16A may be compressed using any suitable compression scheme.For example, the data in tone map portion 16A may be compressed in amanner compatible with JPEG or MPEG standards.

In some embodiments of the invention, HDR information portion 16Bcontains ratios between values specified by tone map portion 16A forindividual pixels and the values specified by original HDR image 12 forthe same pixels. In such embodiments, HDR information 16B may begenerated by dividing the values specified by original HDR image 12 bythe corresponding values specified by tone map portion 16A. The dataresulting from this operation may be stored as HDR information portion16B. The precision with which the data values in HDR information portion16B is represented may be selected to provide acceptable quality inreconstructed HDR images. In some embodiments of the invention the datavalues in HDR information portion 16B are each represented by one byte(8 bits) prior to compression.

In some embodiments of the invention, HDR information portion 16Bspecifies relationships between the luminance of pixels in reconstructedHDR image 21 and the luminance specified for corresponding pixels bytone map information 16A. In such embodiments, HDR information portion16B does not need to contain color information.

HDR information portion 16B may comprise ratios of the luminancespecified by original HDR image 12 for areas or pixels within an imageto the luminance specified by tone map portion 16A for the correspondingareas or pixels. In such embodiments, color information is carried bytone map portion 16A. In such embodiments, HDR portion 16B can have thesame structure as a grey-scale image. For example, where HDR datastructure 16 comprises a JFIF file, HDR portion 16B may be encoded as aJPEG grey-scale image. Where HDR data structure 16 comprises a MPEGfile, HDR portion 16B may be encoded as a MPEG grey-scale image.

FIG. 3 shows an HDR encoder 50 according to an embodiment of theinvention wherein the HDR information used to make HDR portion 16B ofdata structure 16 comprises ratios of pixel values in HDR image 12 tothe corresponding values specified by tone map portion 16A. Encoder 50receives HDR image data 12. Encoder 50 obtains tone map data 15 eitherby extracting tone map data 15 from HDR image data 12, as indicated bydashed line 13 and tone mapper 17, or by receiving tone map data 15 fromsome other source, as indicated by dashed line 13A. Tone mapper 17preferably does not clip colour or luminance values, and maintainscolour and luminance ratios for each pixel in tone map data 15.

In the illustrated embodiment, encoder 50 includes a standard encoder52. Standard encoder encodes tone map data 15 to produce encoded tonemap data 15A. Encoded tone map data 15A can be read with a standardviewer. For example, standard encoder 52 may comprise an encoder thatencodes tone map data 15 as JPEG or MPEG encoded tone map data that canbe read by a JPEG or MPEG viewer. Encoded tone map data is saved intotone map data portion 16A of HDR data structure 16.

In some embodiments of the invention, encoder 50 receives encoded tonemap data 15A from some external source. In such embodiments, encoder 50does not need to incorporate standard encoder 52.

Encoded tone map data 15A is decoded by decoder 54 to yieldreconstructed tone map data 55. HDR image data 12 is divided byreconstructed tone map data 55 by divider 56 to yield ratio data 57.Ratio data 57 is optionally compressed by data compressor 58 to yieldHDR information 16B. Data compressor 58 may conveniently comprise a JPEGor MPEG encoder. In some embodiments of the invention, the same JPEG orMPEG encoder is used to encode both tone map portion 16A and HDRinformation portion 16B of HDR data structure 16.

In some embodiments of the invention, ratio data 57 comprises somefunction of the ratio of values of HDR image data 12 to correspondingvalues specified by tone map data 15 (or tone map portion 16A). Forexample, ratio data 57 may comprise information specifying a logarithmof such a ratio.

In some alternative embodiments of the invention, tone map data 15 isprovided directly to divider 56 as indicated by line 53. In suchembodiments, decoder 54 is not required. Where tone map portion 16A isencoded using a lossy algorithm, such as JPEG or MPEG encoding, it ispreferable to base HDR information portion 16B on reconstructed tone mapdata 55 instead of on tone map data 15. Basing HDR information portion16B on reconstructed tone map data 55 permits a more accuratereconstruction of HDR image data 12 from HDR data structure 16 in caseswhere tone map information portion 16A is encoded by a lossy encodingprocess. Tone map information portion 16A, rather than tone map data 15will be used to reconstruct HDR image 21 (FIG. 1).

Compressor 58 may take any of a number of forms. In some embodiments ofthe invention, compressor 58 performs one or more of the followingoperations:

-   -   downsampling of ratio data 57;    -   compressing ratio data 57.        Any suitable form of compression may be used. In a currently        preferred embodiment of the invention, compressor 58 both        downsamples ratio data 57 and encodes the downsampled ratio        data. Where ratio data 57 is downsampled, HDR information        portion 16B has an image size smaller than an image size of        ratio data 57 or tone map data 15 (i.e. HDR information portion        16B specifies values for a number of pixels that is smaller than        a number of pixels for which ratio data 57 or tone map data 15        specifies values). In such cases, HDR information portion 16B        has a lower spatial resolution than tone map data 15.

In those embodiments of the invention wherein ratio data 57 is subjectedto downsampling, or other lossy compression mechanisms, HDR information16B may lack all of the details necessary to accurately reconstruct HDRimage data 12. Distortions resulting from the lossy compression of ratiodata 57 may be at least partially compensated for by applyingcorrections to tone map portion 16A and/or HDR information portion 16B.

FIG. 4 is a flow chart that illustrates the operation of methods 60which apply corrections to the data in tone map portion 16A or HDRinformation portion 16B to reduce artefacts resulting from lossyencoding of tone map portion 16A and/or HDR information portion 16B.Methods 60 acquire HDR image data 90 and tone map data 91 in blocks 62and 64. HDR image data 90 and tone map data 91 may be obtained in anysuitable manner including those manners described above. In someembodiments, tone map data 91 is extracted from HDR image data 90 asindicated by arrow 65.

In block 66, tone map data 91 is encoded to yield encoded tone map data92. In some embodiments of the invention encoding block 66 comprisesJPEG or MPEG encoding. Subsequently, in block 68, encoded tone map data92 is decoded to yield reconstructed tone map data 94. Reconstructionblock 68 may comprise passing encoded tone map data 92 to a suitabledecoder, such as a JPEG or MPEG decoder in the case that block 66comprises JPEG or MPEG encoding.

Block 70 generates ratio data 96 by applying a function which takes asinputs values from HDR image data 90 (first values) and correspondingvalues from reconstructed tone map data 94 (second values). The functionincludes dividing the first values by the second values or vice versa.In a simple embodiment of the invention, ratio data 96 includes a valueRI for each pixel in an image given by:

$\begin{matrix}{{{RI}\left( {x,y} \right)} = \frac{L\left( {{HDR}\left( {x,y} \right)} \right)}{L\left( {{TM}\left( {x,y} \right)} \right)}} & (1)\end{matrix}$

where: (x, y) are coordinates identifying a pixel; L is a function whichreturns the luminance of the pixel from data for the pixel; HDR(x, y) isthe pixel data in HDR image data 90 at coordinates (x, y); and, TM(x, y)is the pixel data in reconstructed tone map data 94 (or, tone map data91) for the pixel at coordinates (x, y). In some embodiments, the ratiodata stores the logarithm of RI, the square root of RI or anotherfunction of RI.

Blocks 72 and 74 encode ratio data 96. In this example embodiment, theencoding includes downsampling ratio data 96 in block 72 to yielddownsampled ratio data 98 and then compressing downsampled ratio data 98to yield encoded ratio data 100. The amount of downsampling performed inblock 72 may be selected based upon the competing goals of making HDRimage portion 16B small and making a HDR image reconstructed from HDRdata structure 16 reproduce HDR image data 90 with the highest fidelity.In some embodiments of the invention, ratio data 96 is downsampledsufficiently that downsampled ratio data 98 has fewer pixels than ratiodata 96 by a factor in the range of 4 to 15.

For example, downsampling may be performed using a Gaussian filterkernel, which follows a weighting formula of e^(−x̂2/R̂2)), where x is thedistance from the output pixel's centre in the input image and R is adownsampling radius. The downsampling radius may be defined as the areaunder which the weights of the contributing input pixels sum to asignificant portion of the total value of the output pixel.

Any suitable form of data compression may be performed in block 74. Insome embodiments of the invention, block 74 performs JPEG encoding. Inother embodiments of the invention, block 74 performs MPEG encoding.

In block 76, reconstructed ratio data 102 is created by decoding encodedratio data 100. Reconstructed ratio data 102 will typically not beidentical to ratio data 96 because of data loss in blocks 74 and 76.

In block 78 reconstructed HDR image data 104 is created by applying, toreconstructed ratio data 102, the inverse of the function applied inblock 70 to the ratio data and then, for each pixel, multiplying theluminance for the pixel in reconstructed tone map data 94 by the result.For example, where ratio data 96 stores the values RI as defined inEquation (1) then reconstructed HDR image data 104 may be obtained bymultiplying the luminance for each pixel in reconstructed tone map data94 by the corresponding value of RI from reconstructed ratio data 102.For example, where the ratio data stores the natural logarithm valuesln(RI ), reconstructed HDR image data 104 may be obtained by raising e,the base of natural logarithms, to the power of the value inreconstructed ratio data 102 and then multiplying the result by theluminance for each pixel in reconstructed tone map data 94.

Reconstructed HDR image data 104 will differ from original HDR imagedata 90 because reconstructed ratio data 102 is not identical tooriginal ratio data 96 and, usually less importantly, because ofrounding errors in ratio data 96. Block 80 optionally comparesreconstructed HDR image data 104 to original HDR image data 90 todetermine if any correction is required and to determine how thecorrection will be performed. Correction may be performed by correctingthe data of tone map portion 16A and/or by correcting the data of HDRinformation portion 16B. Some methods simply perform one or the other ofthese corrections.

Block 82 obtains corrected tone map data 106. Corrected tone map data106 can be obtained by dividing original HDR image data 90 byreconstructed ratio data 102. Corrected tone map data 106 can then beencoded, if necessary, as indicated by block 83 and stored as tone mapdata portion 16A of HDR data structure 16 in block 84. Thisprecorrection may be performed at any time after reconstructed ratiodata 102 is available. For many purposes, this precorrection does notdegrade significantly the image that can be seen by viewing tone mapdata portion 16A with a conventional image viewer. This correction tendsmake the image represented by tone map portion 16A somewhat sharper thanwould be the case in the absence of this correction. Reconstructed HDRimage data 104 may be stored as HDR information portion 16B of HDR datastructure 16 in block 86.

In some cases it is undesirable to alter the tone map data stored intone map portion 16A. For example, encoded tone map data 92 may havebeen carefully optimized to provide the best image quality when viewedwith a particular viewer, such as, for example, the MPEG decoder in aDVD player. In such cases, encoded tone map data 92 may be stored intone map portion 16A of data structure 16 and ratio data 96 may bestored in HDR information portion 16B of HDR data structure 16.Corrections to the appearance of an HDR image produced from datastructure 16 may be made by correcting HDR information portion 16B uponreconstruction of the HDR image. For example the data in HDR informationportion 16B may be corrected by a viewer capable of processing HDRimages.

FIG. 5 is a flow chart that illustrates the operation of a method 110which applies postcorrection to the data in HDR information portion 16Bto reduce artefacts resulting from lossy encoding of HDR informationportion 16B. Method 110 may be carried out on a processor capable ofprocessing HDR images. Tone map data portion 16A is decoded at block 112by a standard decoder to produce standard image 19. The decoded tone mapinformation is used to correct HDR information portion 16B at block 114.The corrected HDR information is decoded at block 116 by a HDR decoderto produce reconstructed HDR image 21.

In simple cases, where the spatial frequency content of the fullresolution image represented by tone map data portion 16A issubstantially the same as that of ratio data 96 then corrected ratiodata may be obtained by performing the calculation:

$\begin{matrix}{{RI}_{CORRECTED} = {{RI} \times \frac{L({TM})}{L\left( {TM}_{R} \right)}}} & (2)\end{matrix}$

where: RI_(CORRECTED) is the corrected value for RI on which correctedHDR information is based; RI is the ratio for the pixel from ratio data96; L(TM) is the luminance for the pixel from tone map data 91; andL(TM_(R)) is the luminance for a corresponding pixel of tone map datathat has been downsampled in the same manner as performed in block 72 toyield downsampled ratio data 98. The tone map data may be downsampledthen upsampled again, in the same manner as the ratio image RI, so thatTM and TM_(R) have the same resolution.

This simple correction is not always adequate because the spatialfrequencies present in ratio data 96 are not the same as the spatialfrequencies present in tone map data 91 for all images. It is thereforepreferable to include in the correction function a factor that takesinto account variance in the ratio between the values of RI in ratiodata 96 and the corresponding values of L(TMR). One way to take thisvariance into account is to generate corrected values RI_(CORRECTED)according to:

$\begin{matrix}{{RI}_{CORRECTED} = {{RI} \times \left( \frac{L({TM})}{L\left( {TM}_{R} \right)} \right)^{\sigma}}} & (3)\end{matrix}$

where: σ is a measure of the variance in the ratio between the values ofRI in ratio data 96 and the corresponding values of L(TM_(R)). In someembodiments of the invention, σ is computed according to:

$\begin{matrix}{\sigma = \frac{{var}({RI})}{{var}\left( {L\left( {TM}_{R} \right)} \right)}} & (4)\end{matrix}$

The variance function var(x) may be defined as a difference between themaximum and minimum values of x for pixels in a neighborhood, divided byan average value for x in the neighborhood or divided by the value for xfor a pixel located centrally in the neighborhood. For example, thevariance may be computed over a block of pixels centered on a pixel inquestion. The size of the neighborhood over which a is computed ispreferably equal to the downsampling radius for the downsampling ofblock 72.

As the postcorrection provided by block 114 can introduce artefacts, itis desirable to be conservative in selecting the magnitude of thecorrection. For example, where var(L(TMR)) is greater than the errorwhich is sought to be corrected, a may be set to zero. The magnitude ofthe error may be determined by the comparison of block 80 and stored indata structure 16. It is also desirable to ensure that 0≦σ≦1. Allowing σto have values such that σ>1 can result in values of RI_(CORRECTED) thatare undesirably high.

In embodiments of the invention which optionally perform theprecorrection of blocks 82, 83 and 84 and which also permit thecorrection of block 83 to be performed upon viewing an HDR image, it isdesirable to include a flag in data structure 16 that indicates whetheror not precorrection has been performed. The flag is preferably providedin a comment field or an application extension field where it can beignored by standard displays that do not support HDR images.

In some cases, HDR displays are capable of rendering colors that areoutside of the color gamut of a conventional display. It is desirable toprovide a mechanism which permits high fidelity reproduction of thecolors specified by original HDR image data. One way to provide enhancedcolor is to scale color information so that any color having a primarycomponent outside of a range which can be handled effectively by theencoder used to encode tone map portion 16A (which may be, for example,a JPEG or MPEG encoder) is scaled back into the range that can behandled by the encoder. The ratio data can be adjusted to correctlyrecover the scaled color.

One way to provide enhanced color is to apply a global desaturation tothe image while creating tone map portion 16A. The amount ofdesaturation may be chosen to ensure that all colors in the image willbe within the range which can be handled effectively by the JPEG orother encoder used to encode tone map portion 16A. This method ispreferable to the method described above because it is capable ofhandling colors having negative primary components. Negative primarycomponents are allowed in some HDR formats and may be necessary torepresent colors outside of the standard RGB gamut. The desaturationprocess may be reversed during decoding by an HDR viewer.

Input color saturation level may be defined as:

$\begin{matrix}{S \equiv {1 - \frac{\min \left( {R,G,B} \right)}{Y}}} & (5)\end{matrix}$

where: S is the saturation level; R, G and B are values for the red,green and blue primary color components respectively; and Y is theoverall luminance. The saturation level will have a value greater thanone if the image contains any negative values for primary colorcomponents.

Where the saturation level is zero, no additional processing of theimage is needed. Where the saturation level is not zero, the saturationlevel may be modified according to:

S′=α×S ^(β)  (6)

where: α and β are parameters; and S′ is the corrected saturation. The αparameter indicates how much saturation to keep in the encoded colors.

Altering the saturation level may be achieved by deriving new values forthe primary components for each pixel of the image. This is performed insome embodiments according to:

$\begin{matrix}{R^{\prime} = {{\left( {1 - \frac{S^{\prime}}{S}} \right)Y} + {\frac{S^{\prime}}{S}R}}} & (7) \\{G^{\prime} = {{\left( {1 - \frac{S^{\prime}}{S}} \right)Y} + {\frac{S^{\prime}}{S}G}}} & (8) \\{B^{\prime} = {{\left( {1 - \frac{S^{\prime}}{S}} \right)Y} + {\frac{S^{\prime}}{S}B}}} & (9)\end{matrix}$

where R′, G′ and B′ are the scaled values for R, G and B respectively.

Note that this transformation does not alter the luminance, Y. Theprimary component that was smallest prior to the transformation remainssmallest after the transformation. The original color values can berecovered by inverting equations (7), (8) and (9). For example, if theprimary color component having the smallest value for a pixel were blue,then the inverse transformation for the blue channel for that pixelwould be given by:

$\begin{matrix}{B = {Y - {Y \times \left( \frac{Y - B^{\prime}}{\alpha \; Y} \right)^{1/\beta}}}} & (10)\end{matrix}$

and the inverse transformations for the red and green channels would begiven by:

$\begin{matrix}{R = {Y - {\frac{\left( {Y - R^{\prime}} \right)}{\alpha}\left( {1 - \frac{B}{Y}} \right)^{1 - \beta}}}} & (11) \\{G = {Y - {\frac{\left( {Y - G^{\prime}} \right)}{\alpha}\left( {1 - \frac{B}{Y}} \right)^{1 - \beta}}}} & (12)\end{matrix}$

respectively.

EXAMPLES

A number of HDR images were stored in HDR data structures 16, asdescribed above. The original images were compared to HDR imagesreconstructed from HDR data structures 16. Daly's Visual DifferencesPredictor (VDP), as described in Daly, S., The visual differencespredictor: An algorithm for the assessment of image fidelity, In DigitalImages and Human Vision, A. B. Watson editor, MIT Press, CambridgeMass., 1993, was used to evaluate what percentage of pixels in thereconstructed HDR images are likely (e.g. have a probability greaterthan 75%) to be perceived by humans as being different from thecorresponding pixels of the original HDR image under typical viewingconditions. VDP was found to be an excellent predictor of whendifferences could be perceived between images.

A first set of experiments involved using various tone mapping operatorsto produce tone map portion 16A and, for each tone map operator,correcting either tone map portion 16A or HDR information portion 16Baccording to one of the correction methods described above. Tone mapportion 16A and HDR information 16B were each encoded using JPEGencoding at two quality levels, 90 and 100. This set of experimentsyielded the results shown in Table I.

TABLE I Image Quality for Several Tone Mapping Operators Tone MappingJPEG VDP using VDP using Operator Quality precorrection postcorrectionBilateral Filter 90 0.93% 5.4% 100 0.02% 1.8% Reinhard Global 90  2.5%4.7% 100 0.09% 2.8% Histogram Adj. 90  5.9%  21% 100 0.63%  17% Gradient90  7.5%  36% 100  3.0%  34%

The VDP values in Table I are averaged over a number of images. It canbe seen that the selection of a tone mapping operator can have aconsiderable effect upon the quality of the HDR image that can bereconstructed from a HDR data structure 16. Of the tone mappingoperators used in these experiments, the bilateral filter appeared toprovide the best results, on average.

Certain implementations of the invention comprise computer processorswhich execute software instructions which cause the processors toperform a method of the invention. For example, one or more processorsin a computer system may implement the methods of any of FIGS. 1 to 5 byexecuting software instructions in a program memory accessible to theprocessors. The invention may also be provided in the form of a programproduct. The program product may comprise any medium which carries a setof computer-readable signals comprising instructions which, whenexecuted by a computer processor, cause the data processor to execute amethod of the invention. Program products according to the invention maybe in any of a wide variety of forms. The program product may comprise,for example, physical media such as magnetic data storage mediaincluding floppy diskettes, hard disk drives, optical data storage mediaincluding CD ROMs, DVDs, electronic data storage media including ROMs,flash RAM, or the like or transmission-type media such as digital oranalog communication links. The instructions may optionally be presentin the computer-readable signals in a compressed and/or encryptedformat.

Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to above, unless otherwise indicated,reference to that component (including a reference to a “means”) shouldbe interpreted as including as equivalents of that component anycomponent which performs the function of the described component (i.e.,that is functionally equivalent), including components which are notstructurally equivalent to the disclosed structure which performs thefunction in the illustrated exemplary embodiments of the invention.

As will be apparent to those skilled in the art in the light of theforegoing disclosure, many alterations and modifications are possible inthe practice of this invention without departing from the spirit orscope thereof. Accordingly, the scope of the invention is to beconstrued in accordance with the substance defined by the followingclaims.

What is claimed is:
 1. A decoding method comprising: receiving a JPEGcompressed file, the JPEG compressed file includes a low dynamic range(LDR) image and a ratio image, the ratio image separated as anapplication extension from the LDR image; and reconstructing areconstructed HDR image, the reconstructed HDR image determined by theLDR image and the ratio image, wherein LDR image has a lower dynamicrange than the reconstructed HDR image, and wherein a pixel value of theratio image is from a logarithm of a ratio of a luminance of a pixel ofan HDR image and a luminance of a corresponding pixel of the LDR image.